{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 关键词搜索\n",
    "\n",
    "Kaggle竞赛题：https://www.kaggle.com/c/home-depot-product-search-relevance\n",
    "\n",
    "鉴于课件里已经完整的show了NLTK在各个NLP处理上的用法，我这里就不再重复使用了。\n",
    "\n",
    "本篇的教程里会尽量用点不一样的库，让大家感受一下Python NLP领域各个库的优缺点。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step1：导入所需"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "所有要用到的库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.ensemble import RandomForestRegressor, BaggingRegressor\n",
    "from nltk.stem.snowball import SnowballStemmer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "读入训练/测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_train = pd.read_csv('../input/train.csv', encoding=\"ISO-8859-1\")\n",
    "df_test = pd.read_csv('../input/test.csv', encoding=\"ISO-8859-1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里还有个有用的玩意儿，叫产品介绍"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_desc = pd.read_csv('../input/product_descriptions.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "看看数据们都长什么样子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>product_uid</th>\n",
       "      <th>product_title</th>\n",
       "      <th>search_term</th>\n",
       "      <th>relevance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>100001</td>\n",
       "      <td>Simpson Strong-Tie 12-Gauge Angle</td>\n",
       "      <td>angle bracket</td>\n",
       "      <td>3.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>100001</td>\n",
       "      <td>Simpson Strong-Tie 12-Gauge Angle</td>\n",
       "      <td>l bracket</td>\n",
       "      <td>2.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>100002</td>\n",
       "      <td>BEHR Premium Textured DeckOver 1-gal. #SC-141 ...</td>\n",
       "      <td>deck over</td>\n",
       "      <td>3.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16</td>\n",
       "      <td>100005</td>\n",
       "      <td>Delta Vero 1-Handle Shower Only Faucet Trim Ki...</td>\n",
       "      <td>rain shower head</td>\n",
       "      <td>2.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17</td>\n",
       "      <td>100005</td>\n",
       "      <td>Delta Vero 1-Handle Shower Only Faucet Trim Ki...</td>\n",
       "      <td>shower only faucet</td>\n",
       "      <td>2.67</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  product_uid                                      product_title  \\\n",
       "0   2       100001                  Simpson Strong-Tie 12-Gauge Angle   \n",
       "1   3       100001                  Simpson Strong-Tie 12-Gauge Angle   \n",
       "2   9       100002  BEHR Premium Textured DeckOver 1-gal. #SC-141 ...   \n",
       "3  16       100005  Delta Vero 1-Handle Shower Only Faucet Trim Ki...   \n",
       "4  17       100005  Delta Vero 1-Handle Shower Only Faucet Trim Ki...   \n",
       "\n",
       "          search_term  relevance  \n",
       "0       angle bracket       3.00  \n",
       "1           l bracket       2.50  \n",
       "2           deck over       3.00  \n",
       "3    rain shower head       2.33  \n",
       "4  shower only faucet       2.67  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product_uid</th>\n",
       "      <th>product_description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100001</td>\n",
       "      <td>Not only do angles make joints stronger, they ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>100002</td>\n",
       "      <td>BEHR Premium Textured DECKOVER is an innovativ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>100003</td>\n",
       "      <td>Classic architecture meets contemporary design...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>100004</td>\n",
       "      <td>The Grape Solar 265-Watt Polycrystalline PV So...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>100005</td>\n",
       "      <td>Update your bathroom with the Delta Vero Singl...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   product_uid                                product_description\n",
       "0       100001  Not only do angles make joints stronger, they ...\n",
       "1       100002  BEHR Premium Textured DECKOVER is an innovativ...\n",
       "2       100003  Classic architecture meets contemporary design...\n",
       "3       100004  The Grape Solar 265-Watt Polycrystalline PV So...\n",
       "4       100005  Update your bathroom with the Delta Vero Singl..."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_desc.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "看来不要做太多的复杂处理，我们于是直接合并测试/训练集，以便于统一做进一步的文本预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_all = pd.concat((df_train, df_test), axis=0, ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>product_title</th>\n",
       "      <th>product_uid</th>\n",
       "      <th>relevance</th>\n",
       "      <th>search_term</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>Simpson Strong-Tie 12-Gauge Angle</td>\n",
       "      <td>100001</td>\n",
       "      <td>3.00</td>\n",
       "      <td>angle bracket</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>Simpson Strong-Tie 12-Gauge Angle</td>\n",
       "      <td>100001</td>\n",
       "      <td>2.50</td>\n",
       "      <td>l bracket</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>BEHR Premium Textured DeckOver 1-gal. #SC-141 ...</td>\n",
       "      <td>100002</td>\n",
       "      <td>3.00</td>\n",
       "      <td>deck over</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16</td>\n",
       "      <td>Delta Vero 1-Handle Shower Only Faucet Trim Ki...</td>\n",
       "      <td>100005</td>\n",
       "      <td>2.33</td>\n",
       "      <td>rain shower head</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17</td>\n",
       "      <td>Delta Vero 1-Handle Shower Only Faucet Trim Ki...</td>\n",
       "      <td>100005</td>\n",
       "      <td>2.67</td>\n",
       "      <td>shower only faucet</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                      product_title  product_uid  \\\n",
       "0   2                  Simpson Strong-Tie 12-Gauge Angle       100001   \n",
       "1   3                  Simpson Strong-Tie 12-Gauge Angle       100001   \n",
       "2   9  BEHR Premium Textured DeckOver 1-gal. #SC-141 ...       100002   \n",
       "3  16  Delta Vero 1-Handle Shower Only Faucet Trim Ki...       100005   \n",
       "4  17  Delta Vero 1-Handle Shower Only Faucet Trim Ki...       100005   \n",
       "\n",
       "   relevance         search_term  \n",
       "0       3.00       angle bracket  \n",
       "1       2.50           l bracket  \n",
       "2       3.00           deck over  \n",
       "3       2.33    rain shower head  \n",
       "4       2.67  shower only faucet  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "合并之后我们得到:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(240760, 5)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "产品介绍也是一个极有用的信息，我们把它拿过来："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df_all = pd.merge(df_all, df_desc, how='left', on='product_uid')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>product_title</th>\n",
       "      <th>product_uid</th>\n",
       "      <th>relevance</th>\n",
       "      <th>search_term</th>\n",
       "      <th>product_description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>Simpson Strong-Tie 12-Gauge Angle</td>\n",
       "      <td>100001</td>\n",
       "      <td>3.00</td>\n",
       "      <td>angle bracket</td>\n",
       "      <td>Not only do angles make joints stronger, they ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>Simpson Strong-Tie 12-Gauge Angle</td>\n",
       "      <td>100001</td>\n",
       "      <td>2.50</td>\n",
       "      <td>l bracket</td>\n",
       "      <td>Not only do angles make joints stronger, they ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>BEHR Premium Textured DeckOver 1-gal. #SC-141 ...</td>\n",
       "      <td>100002</td>\n",
       "      <td>3.00</td>\n",
       "      <td>deck over</td>\n",
       "      <td>BEHR Premium Textured DECKOVER is an innovativ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16</td>\n",
       "      <td>Delta Vero 1-Handle Shower Only Faucet Trim Ki...</td>\n",
       "      <td>100005</td>\n",
       "      <td>2.33</td>\n",
       "      <td>rain shower head</td>\n",
       "      <td>Update your bathroom with the Delta Vero Singl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17</td>\n",
       "      <td>Delta Vero 1-Handle Shower Only Faucet Trim Ki...</td>\n",
       "      <td>100005</td>\n",
       "      <td>2.67</td>\n",
       "      <td>shower only faucet</td>\n",
       "      <td>Update your bathroom with the Delta Vero Singl...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                      product_title  product_uid  \\\n",
       "0   2                  Simpson Strong-Tie 12-Gauge Angle       100001   \n",
       "1   3                  Simpson Strong-Tie 12-Gauge Angle       100001   \n",
       "2   9  BEHR Premium Textured DeckOver 1-gal. #SC-141 ...       100002   \n",
       "3  16  Delta Vero 1-Handle Shower Only Faucet Trim Ki...       100005   \n",
       "4  17  Delta Vero 1-Handle Shower Only Faucet Trim Ki...       100005   \n",
       "\n",
       "   relevance         search_term  \\\n",
       "0       3.00       angle bracket   \n",
       "1       2.50           l bracket   \n",
       "2       3.00           deck over   \n",
       "3       2.33    rain shower head   \n",
       "4       2.67  shower only faucet   \n",
       "\n",
       "                                 product_description  \n",
       "0  Not only do angles make joints stronger, they ...  \n",
       "1  Not only do angles make joints stronger, they ...  \n",
       "2  BEHR Premium Textured DECKOVER is an innovativ...  \n",
       "3  Update your bathroom with the Delta Vero Singl...  \n",
       "4  Update your bathroom with the Delta Vero Singl...  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "好了，现在我们得到一个全体的数据大表格"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: 文本预处理\n",
    "\n",
    "我们这里遇到的文本预处理比较简单，因为最主要的就是看关键词是否会被包含。\n",
    "\n",
    "所以我们统一化我们的文本内容，以达到任何term在我们的数据集中只有一种表达式的效果。\n",
    "\n",
    "我们这里用简单的Stem做个例子：\n",
    "\n",
    "（有兴趣的同学可以选用各种你觉得靠谱的预处理方式：去掉停止词，纠正拼写，去掉数字，去掉各种emoji，等等）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stemmer = SnowballStemmer('english')\n",
    "\n",
    "def str_stemmer(s):\n",
    "    return \" \".join([stemmer.stem(word) for word in s.lower().split()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了计算『关键词』的有效性，我们可以naive地直接看『出现了多少次』"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def str_common_word(str1, str2):\n",
    "    return sum(int(str2.find(word)>=0) for word in str1.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，把每一个column都跑一遍，以清洁所有的文本内容"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_all['search_term'] = df_all['search_term'].map(lambda x:str_stemmer(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_all['product_title'] = df_all['product_title'].map(lambda x:str_stemmer(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_all['product_description'] = df_all['product_description'].map(lambda x:str_stemmer(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: 自制文本特征\n",
    "\n",
    "一般属于一种脑洞大开的过程，想到什么可以加什么。\n",
    "\n",
    "当然，特征也不是越丰富越好，稍微靠谱点是肯定的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 关键词的长度："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_all['len_of_query'] = df_all['search_term'].map(lambda x:len(x.split())).astype(np.int64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 标题中有多少关键词重合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df_all['commons_in_title'] = df_all.apply(lambda x:str_common_word(x['search_term'],x['product_title']), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 描述中有多少关键词重合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_all['commons_in_desc'] = df_all.apply(lambda x:str_common_word(x['search_term'],x['product_description']), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "等等等等。。变着法子想出些数字能代表的features，一股脑放进来~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "搞完之后，我们把不能被『机器学习模型』处理的column给drop掉"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_all = df_all.drop(['search_term','product_title','product_description'],axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: 重塑训练/测试集\n",
    "\n",
    "舒淇说得好，要把之前脱下的衣服再一件件穿回来\n",
    "\n",
    "数据处理也是如此，搞完一圈预处理之后，我们让数据重回原本的样貌"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 分开训练和测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_train = df_all.loc[df_train.index]\n",
    "df_test = df_all.loc[df_test.index]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 记录下测试集的id\n",
    "\n",
    "留着上传的时候 能对的上号"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_ids = df_test['id']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 分离出y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train = df_train['relevance'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 把原集中的label给删去\n",
    "\n",
    "否则就是cheating了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train = df_train.drop(['id','relevance'],axis=1).values\n",
    "X_test = df_test.drop(['id','relevance'],axis=1).values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: 建立模型\n",
    "\n",
    "我们用个最简单的模型：Ridge回归模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用CV结果保证公正客观性；并调试不同的alpha值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "params = [1,3,5,6,7,8,9,10]\n",
    "test_scores = []\n",
    "for param in params:\n",
    "    clf = RandomForestRegressor(n_estimators=30, max_depth=param)\n",
    "    test_score = np.sqrt(-cross_val_score(clf, X_train, y_train, cv=5, scoring='neg_mean_squared_error'))\n",
    "    test_scores.append(np.mean(test_score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "画个图来看看："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhIAAAFyCAYAAACgITN4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3Xu8lWP+//HXp7OKzYhyiHH6OQyjg8M4DJMkoZxlJyIq\n5DA55JxxZnxpGBqMIQabMENUIsZkhhidkAjjTKlBDoUOn98fn7W1Wvbe7XW3d/daa7+fj8f92Na9\n7vven7uy13tf13Vfl7k7IiIiIkk0SrsAERERKV4KEiIiIpKYgoSIiIgkpiAhIiIiiSlIiIiISGIK\nEiIiIpKYgoSIiIgkpiAhIiIiiSlIiIiISGIKEiIiIpKYgoRIHsysn5ktzdoWmtmbZvZHM1s37fqK\nkZkdbGZjzWyumX1vZh+b2QNm1iXz/g2ZP+tNa7jGFZljtq3hmDtz/u6ytwX1cW8iDUGTtAsQKUIO\nXAS8B7QAdgdOAnqY2bbu/l2KtRUVM7sT6AdMAa4DZgPrAQcDE8xsN+Be4FSgD3B5NZc6Epju7q+t\n4Ft+BxwPWM7+JYluQEQUJEQSesLdp2T++w4z+xwYAhwIPLAyFzaz5sAPXuIr6pnZWUSIuN7dz8p5\n+yozOwpY7O4vm9nbQDlVBAkz2wXYBBhai2+72N0rEtTa0t2rbLWo6b26uL5IoVPXhkjdeIb4LXcT\nADNby8z+z8xeMbOvzWx+pvn+l9knmdmemab13mZ2uZl9BHwLrJ7gGkeY2aWZroGvzexBM1vDzJqb\n2R/MbE5m/1/MrGlNN5PpqvnazFpU8V6FmX1iZpZ5vYOZjc90TSwws/+a2V9WcP0WwLnA68DZVR3j\n7ve6+8uZl/cCW5lZhyoO7QMsBe6v6XvWVlb31R5mNsLM5gAfZt77Xea9rc3svkyAfC7r3L3M7Dkz\n+8bMvjCzR8xsq5zr13gNkWKjFgmRurF55uv/Ml83BXoBDwLvAm2BQcCzZraNu8/OOf8i4HvgWqA5\n8APwizyvcR4RQq7M1HMqsIj4/7wlcDHwK+C4zPWq6yaAaFU5GdgfeLhyp5mtBhwA3OHubmbrAOOB\nz4CrgC+BnwOH1HBtiO6gnxGtEbVpebk3U38fYFpWPY2Aw4GJ7v5RLa6Dma1dxe4f3P3rnH0jiPu6\nBGiV2VdZ64PALOLPvDJQ7Q2MBd7J1LoacBrwLzPr5O4frOgaIkXJ3bVp01bLjWiKXwJ0AdYGNgB6\nA3OBb4D1Msc1reLcjYCFwAVZ+/Ykfpt+C2iWc3y+15gONM7af29m/+M51/g38N9a3OuHwKicfYdn\n7n/XzOsDM6875vnneGrmvF55nPMi8H7Ovu6Zezy+FuffmTm2qm1szt/xUuBZwHKucXHmvb9Wcf2p\nwKdAWda+7YDFwJ21uYY2bcW4qUVCJH8GPJ312omBl+Xu/imAuy/68eD4rXlNYAHwJtCpimuOdPcf\nsnckuMZd7p49aPBFYhDiHTnHvQicamaN3H1p9bfJg8DAnP773sDH7v585vWXxJ9HLzN71d0X13C9\nbGtkvua2AtTkHuAPZraHu0/M7OtDtOQ8VMtrLCRaVHJbAOblvHbgz+5eVWuJA7dm7zCzdsD2wNXu\nPv/HA91fNbOngP1WdA2RYqUxEiL5c+Ipjb2B3wDbuPtm7j6h8gALQ8xsFvFBN49oJt8OKKvimu/l\n7khwjQ9zXs+vYX+jaq6R7QGiS6RXpp5WQA9gVOUB7v5P4kN8GDAvMybgWDNrtoJrf5X5uvoKjst2\nP/GbfJ9MPc2Bg4jWhPk1nZhlibv/w92fydleqeLY92q4zrs5rzfOfJ1VxbEzgTaZbqGariFSlBQk\nRJL5T+YDaKK7v1nF+xcQjzM+CxwF7EMEj9ep+v+7hXVwjeoeYaxuf4398u7+IvFhekRmVy/icdcH\nco47AtgF+COwPtEC8rKZtazh8m9kvv92NdWQ833mAk8Bh5pZ40w9rYkunPpQ1d9Jbd6ri+uLFA11\nbYjUj0OBZ9x9YPZOM1uTGE+xqq6xskYBp5lZa6Jb4z13/0/uQe7+EvAScJGZlRMf7lV1q1T6F/AF\nUG5mV1bThVCVe4lxEfsRj4N+BTyex/3Ul/czX7es4r2tgHnuruAgJUktEiL1Ywk5v/Gb2eHE4MxV\neY2V9QDxFMmxxAf4cq0RmVCTa3rma/PqLpr5UL0G2Ab4fVXHmNlRZrZDzu5HiN/kTya6WR7OHVuS\nBo8naKYB/cyscvwHFjNt7gOMSas2kfqmFgmR/NXmUb3Hid/O7wCeJ5rwjyIeDayturjGSj1W6O5T\nzewd4AqgGVnjIzL6mdnJwN8zda0ODCDGYYxdweWvJYLEGRbTYT9EzGzZjhj7sCOwa04935rZI8Q4\nCQfuy/OWmlhMdFWVv2W1GiT5czubuOdJmXk0WgKnEC0vlyS4nkhRUJAQyV9tmuGvJD5I+hBjDCYT\nzfFXV3F+dderi2vUxeyYDwDnA2+5+7Sc9/5JfOD3Jua5mE88FdLH3d+nBpnujGPN7FFgIHAm8TTH\nPOIR1aGZcRq57iW6NT5x92fyvJfmwN3VvPcckDvXQ625+9Nmti8RGi4h5vB4Fjh3RX8WIsXMat81\nKSIiIrK8RGMkzGywmb1rsfLhJDPbsYZjK6fvzd6WWDUrJZrZkZlj/pakNhEREVl18g4SZtabeCTt\nYqAjMbBqvJm1qeE0B7Yg+j7bEbP/fVbFtX9O9JtOzH1PRERECk+SFokhwK3ufre7vwGcSMy2138F\n5811988qt9w3MzP33UNMbKOJWkRERIpAXkEis2JgZ7KmB84MmJpATEhT7anAtMyKgU+a2a5VHHMx\nMMfd78ynJhEREUlPvk9ttAEaA3Ny9s+h6olYIBaxGQS8TIyYHkCsXrhT5QhwM9udWJFw+9oWklnB\nrzsx8953tb8FERGRBq8FsVLveHf/3wqOrVG9P/7p7rNYfv75SWa2GdFF0i8zY97dwAB3/yKPS3en\n/qbGFRERaQiOIv/5WJaTb5CYR8y21zZnf1tiIpnaegnYLfPfmxEL3jxmZpWTwDQCMLMfgC3dvaox\nE+8B3HPPPWy99dZ5fOvCNGTIEIYPH552GXVG91O4SuleQPdTyErpXqC07mfmzJn07dsXal6crlby\nChLuvsjMJgNdgdEQKxRmXt+Yx6U6EF0eEIv35C7ccwWxGM9p/HTlwkrfAWy99dZ06lTVisrFpays\nrCTuo5Lup3CV0r2A7qeQldK9QOndT8ZKDw1I0rVxPTAyEyheIrooWgIjAczsKmB9d++XeX068RTG\nDKJPZgDQBegG4O7fE6sZ/sjMvoy3fGaC+kRERGQVyTtIuPuozJwRlxJdGtOA7pklfiHmiWifdUoz\nYt6J9YnHRF8Burq75ooQEREpcokGW7r7CGBENe8dl/P6WmKSqXyuf9yKjxIREZG0aRnxAlFeXp52\nCXVK91O4SuleQPdTyErpXqD07qeuFO2iXWbWCZg8efLkUhz8IiIiUm+mTJlC586dATq7+5SVuZZa\nJERERCQxBQkRERFJTEFCREREElOQEBERkcQUJERERCQxBQkRERFJTEFCREREElOQEBERkcQUJERE\nRCQxBQkRERFJTEFCREREElOQEBERkcQUJERERCQxBQkRERFJTEFCREREElOQEBERkcQUJERERCQx\nBQkRERFJTEFCREREElOQEBERkcQUJERERCQxBQkRERFJTEFCREREElOQEBERkcQUJERERCQxBQkR\nERFJTEFCREREElOQEBERkcQUJERERCSxog8SDz+cdgUiIiINV9EHiauugtGj065CRESkYSr6INGl\nCxx5JEyalHYlIiIiDU/RB4nLL4fOnaFnT3jrrbSrERERaViKPkg0bw6PPgrrrAP77gtz5qRdkYiI\nSMNR9EEC4Gc/g3HjYOFCOOAA+OabtCsSERFpGEoiSABsvDGMHQtvvAG9e8PixWlXJCIiUvpKJkgA\ndOgAf/sbPPkknHgiuKddkYiISGkrqSAB0K0b/OUvsV16adrViIiIlLYmaRdQH445Bj76CC64ADbc\nEI4/Pu2KRERESlOiFgkzG2xm75rZQjObZGY71nDsnma2NGdbYmbrZh1zsJn9x8y+MLNvzGyqmfVN\nUlul886L7o1Bg2LshIiIiNS9vIOEmfUGrgMuBjoC04HxZtamhtMc2AJol9nWc/fPst7/H3A58Ctg\nO+BO4E4z2yff+pbVCTfdBPvvD4cfDi+/nPRKIiIiUp0kLRJDgFvd/W53fwM4EVgA9F/BeXPd/bPK\nLfsNd5/o7o+6+5vu/q673wi8AuyWoL4fNW4MFRXwy19GoPjvf1fmaiIiIpIrryBhZk2BzsDTlfvc\n3YEJwC41nQpMM7NPzOxJM9t1Bd+nK7B59vdJqmVLeOwxKCuLCavmzVvZK4qIiEilfFsk2gCNgdz5\nI+cQXRZV+RQYBBwKHAJ8CDxrZh2yDzKzNczsazP7ARgDnOHuE/Osr+qi28ATT8D8+TGV9oIFdXFV\nERERqfenNtx9FjAra9ckM9uM6CLpl7X/a2B7oDXQFbjRzD5x93E1XX/IkCGUlZUtt6+8vJzy8vLl\n9m26KYwZA7/5DfTpE8uPN26c9K5ERESKQ0VFBRUVFcvtmz9/fp1d3zyPWZsyXRsLgEPdfXTW/pFA\nmbsfXMvr/B7Yzd2rHQNhZn8GNnb3KgdcmlknYPLkyZPp1KlTre9h3LholRg4EG6+OQZlioiINCRT\npkyhc+fOAJ3dfcrKXCuvrg13XwRMJloMADAzy7x+Po9LdSC6PFZUW523GfToAbfdBn/6E1x9dV1f\nXUREpGFJ0rVxPTDSzCYDLxFdFC2BkQBmdhWwvrv3y7w+HXgXmAG0AAYAXYBulRc0s3OBl4F3gObA\n/kBfYGCSm1qR/v3hww/h/PNjwqqjj66P7yIiIlL68g4S7j4qM2fEpUBbYBrQ3d3nZg5pB7TPOqUZ\nMe/E+kS3yCtA15yBlK2Am4ENgYXAG8BR7v5QvvXV1rBhMftl//7Qrl1MrS0iIiL5yWuMRCFJOkYi\n2+LFcOCB8NxzMHFiLPolIiJS6lIbI1FqmjSBBx6ALbeMsRPvv592RSIiIsWlQQcJgNat47HQli1j\nwqrPP0+7IhERkeLR4IMEwLrrxoRV8+ZFV8d336VdkYiISHFQkMjYYgt4/HGYPBn69oUlS9KuSERE\npPApSGTZeecYM/H3v8MZZ0CRjkMVERFZZRQkcvTsCSNGwI03wnXXpV2NiIhIYav3tTaK0aBBMWHV\n2WfDBhtAzrIdIiIikqEgUY3LLosJq/r1iwmrunRJuyIREZHCo66NapjBn/8cAeKgg+DVV9OuSERE\npPAoSNSgaVN46KFYgrxHj+juEBERkWUUJFZg9dVh7NiYBbNHD/jyy7QrEhERKRwKErWw3nowbhx8\n8kl0c3z/fdoViYiIFAYFiVraemt47DGYNCkGYC5dmnZFIiIi6VOQyMNuu8F998GoUTB0aNrViIiI\npE9BIk+HHAI33BCTVd1wQ9rViIiIpEvzSCRw6qnxBMeQITFh1WGHpV2RiIhIOhQkErr66piwqm9f\naNsWfv3rtCsSERFZ9dS1kVCjRnDnnbDrrrH0+MyZaVckIiKy6ilIrITmzWOl0A03hH33jcdDRURE\nGhIFiZVUVhYTVi1dCvvtB199lXZFIiIiq46CRB3YcEN44gl47z049FD44Ye0KxIREVk1FCTqyC9+\nAY8+ChMnwgkngHvaFYmIiNQ/BYk6tOeecPfd8Ne/wgUXpF2NiIhI/dPjn3Wsd2/4+GM480xo3x5O\nOintikREROqPgkQ9OOOMmLDqlFNg/fXj8VAREZFSpK6NenLddTHw8sgj4YUX0q5GRESkfihI1JNG\njWK8xE47Qc+eMGtW2hWJiIjUPQWJetSiBTzySEyhve++MHt22hWJiIjULQWJerbWWjBuHHz/PRxw\nAHzzTdoViYiI1B0FiVVgo41i9stZs+Dww2HRorQrEhERqRsKEqvI9tvHuhxPPw2DBmnCKhERKQ0K\nEqtQ166xYuidd8Lvfpd2NSIiIitP80isYkcdBR99BOeeG2t0DBiQdkUiIiLJKUikYOjQmLDqpJNi\nwqr990+7IhERkWTUtZECM7jhhphf4ogj4D//SbsiERGRZBQkUtK4Mdx3XwzC3H9/ePvttCsSERHJ\nn4JEilZbDR57LOaa6NED5s5NuyIREZH8KEikbO214Ykn4OuvY8Kqb79NuyIREZHaU5AoAJtsAmPG\nwIwZscjX4sVpVyQiIlI7ChIFonNneOihmE578GBNWCUiIsUhUZAws8Fm9q6ZLTSzSWa2Yw3H7mlm\nS3O2JWa2btYxJ5jZRDP7PLM9VdM1S9W++8Ltt8Ntt8GVV6ZdjYiIyIrlPY+EmfUGrgMGAi8BQ4Dx\nZvb/3H1eNac58P+Ar3/c4f5Z1vt7AvcBzwPfAecCT5rZNu7+ab41FrNjj405Ji68MCas6tcv7YpE\nRESql2RCqiHAre5+N4CZnQjsD/QHfl/DeXPd/auq3nD3o7Nfm9kJwKFAV+CeBDUWtQsvjDBxwgmw\n3nqwzz5pVyQiIlK1vLo2zKwp0Bl4unKfuzswAdilplOBaWb2iZk9aWa7ruBbtQKaAp/nU1+pMIMR\nI6Kr49BDYcqUtCsSERGpWr5jJNoAjYE5OfvnAO2qOedTYBDRwnAI8CHwrJl1qOH7XAN8TASUBqlJ\nE7j/fthmm5iw6r330q5IRETkp+r9qQ13n+Xuf3b3qe4+yd2PJ8ZCDKnqeDM7FzgCOMjdf6jv+gpZ\nq1YxYVWrVtE68b//pV2RiIjI8vIdIzEPWAK0zdnfFpidx3VeAnbL3WlmZwFDga7uPqM2FxoyZAhl\nZWXL7SsvL6e8vDyPcgrXuuvGhFW77gq9esGECTEjpoiISG1UVFRQUVGx3L758+fX2fXN85ywwMwm\nAS+6++mZ1wZ8ANzo7tfW8hpPAl+5+2FZ+4YC5wH7uPsKl7Eys07A5MmTJ9OpU6e87qEYvfQSdOkC\n3bvDgw/GWh0iIiJJTJkyhc6dOwN0dveVGomXpGvjemCAmR1jZlsBtwAtgZEAZnaVmd1VebCZnW5m\nvcxsMzP7hZn9AegC3JR1zDnApcSTHx+YWdvM1irxnZWYnXaCBx6ARx+F3/5WE1aJiEhhyPvxT3cf\nZWZtiA/+tsA0oLu7Vy451Q5on3VKM2LeifWBBcArRNfFxKxjTiSe0ngo59tdkvk+QqzFccstMHAg\ntG8PQ4emXZGIiDR0SeaRwN1HACOqee+4nNfXAjV2ebj7JknqaIgGDIg5Js45Jyas6tMn7YpERKQh\nSxQkJF2XXAIffRSzYLZtC127pl2RiIg0VFq0qwiZwa23RoA45BB45ZW0KxIRkYZKQaJINW0aT29s\nvjn06AEffJB2RSIi0hApSBSx1q1hzBho1izCxBdfpF2RiIg0NAoSRa5du5iwas4cOOgg+O67tCsS\nEZGGREGiBGy5ZUyl/dJLsez40qVpVyQiIg2FgkSJ2GUXqKiIcRNnnZV2NSIi0lAoSJSQgw6CP/4R\nhg+PTUREpL5pHokSM3hwTFh1xhmwwQZwxBFpVyQiIqVMQaIEXXllTFh19NExYdWee6ZdkYiIlCp1\nbZSgRo3gjjvg17+O7o4ZtVqQXUREJH8KEiWqWTN4+GHYaKOYY+Ljj9OuSERESpGCRAkrK4Nx4+K/\n99sP5s9Ptx4RESk9ChIlbv31Y8KqDz6AQw+FH35IuyIRESklChINwDbbwKOPwnPPQf/+mrBKRETq\njoJEA7HHHnDPPXDffXD++WlXIyIipUKPfzYghx8egy6HDIH27WPOCRERkZWhINHA/Pa3MWHVqafG\n+ImDD067IhERKWbq2miArr02Zrzs0weefz7takREpJgpSDRAjRrBXXfBzjtDz57w5ptpVyQiIsVK\nQaKBat4cHnkE1lsP9t0XZs9OuyIRESlGChIN2JprxoRVixbFhFVff512RSIiUmwUJBq49u0jTLzz\nTjzVsWhR2hWJiEgxUZAQttsuujmeeQYGDAD3tCsSEZFioSAhAHTpEgMw77oLhg1LuxoRESkWmkdC\nflReDh99BEOHwoYbwqBBaVckIiKFTkFClnPWWTFh1cknx4RVPXumXZGIiBQydW3Icsxg+HA46CDo\n3RtefDHtikREpJApSMhPNG4cC3x16gQHHABvvZV2RSIiUqgUJKRKq60Go0dDmzbQowd89lnaFYmI\nSCFSkJBq/exnMcfEt99Gy8S336ZdkYiIFBoFCanRz38OY8fCzJmx0NfixWlXJCIihURBQlaoY0d4\n+GF48kk46SRNWCUiIssoSEit7LMP3H57bJddlnY1IiJSKDSPhNRav34xYdWFF8YaHccdl3ZFIiKS\nNgUJycv558eEVQMGLFuCXEREGi51bUhezOCmm2LZ8cMOg8mT065IRETSpCAheWvSBO6/H7bdFvbf\nH959N+2KREQkLQoSkkjLlvDYY7D66tG9MW9e2hWJiEgaFCQksXXWgSeegC+/hF69YMGCtCsSEZFV\nTUFCVspmm8Hjj8P06XDUUbBkSdoViYjIqpQoSJjZYDN718wWmtkkM9uxhmP3NLOlOdsSM1s365ht\nzOyhzDWXmtlpSeqSdOy4Izz4YHR1nHaaJqwSEWlI8g4SZtYbuA64GOgITAfGm1mbGk5zYAugXWZb\nz92zl4FqCbwDnAN8mm9Nkr799oNbb4URI+Caa9KuRkREVpUk80gMAW5197sBzOxEYH+gP/D7Gs6b\n6+5fVfWGu78MvJy5nj6GitTxx8ccE+edB+3awbHHpl2RiIjUt7yChJk1BToDV1buc3c3swnALjWd\nCkwzsxbAa8Dv3P35BPVKgbv4Ypg9G/r3h6VL46uIiJSufFsk2gCNgTk5++cAW1ZzzqfAIKLFoTkw\nAHjWzHZy92l5fn8pcGbRvdGoUbRQLF4MAwemXZWIiNSXep8i291nAbOydk0ys82ILpJ+K3v9IUOG\nUFZWtty+8vJyysvLV/bSklCjRnDzzTFx1aBBESZOPjntqkREGqaKigoqKiqW2zd//vw6u36+QWIe\nsARom7O/LTA7j+u8BOyW5/eu0vDhw+nUqVNdXErqkBnccEOEicGDI0ycpmdxRERWuap+uZ4yZQqd\nO3euk+vnFSTcfZGZTQa6AqMBzMwyr2/M41Id0NMZJc8MrrsOmjaF00+HRYvgzDPTrkpEROpSkq6N\n64GRmUDxEtFF0RIYCWBmVwHru3u/zOvTgXeBGUALYoxEF6Bb5QUzgzi3IQZlNgM2MLPtgW/c/Z1E\ndyYFwQyuvjrCxFlnRZg499y0qxIRkbqSd5Bw91GZOSMuJbo0pgHd3X1u5pB2QPusU5oR806sDywA\nXgG6uvvErGPWB6YS800AnJXZ/gnslW+NUljM4LLLIkycd16EiYsuSrsqERGpC4kGW7r7CGBENe8d\nl/P6WuDaFVzvfTRdd0kzi0dDmzSBCy+MMRO/+13sFxGR4lXvT22IZLvggmiZOOecCBOXX64wISJS\nzBQkZJUbOjTCxBlnRDfHNdcoTIiIFCsFCUnFkCHRzXHaaREmrr9eYUJEpBgpSEhqTj01wsTJJ0c3\nx403KkyIiBQbBQlJ1UknLZsBc9GiZdNri4hIcVCQkNQNGBBjJvr3j5aJ225TmBARKRYKElIQjj02\nWib69YuWiTvugMaN065KRERWREFCCkbfvhEm+vaNlom77orXIiJSuPRjWgrKkUdGeCgvhyVL4K9/\njW4PEREpTAoSUnAOOyy6NXr3jpaJ++6DZs3SrkpERKqiIW1SkA4+GB5+GB57DI44Ar7/Pu2KRESk\nKgoSUrB69oRHHoEnnoBDD4Xvvku7IhERyaUgIQWtRw8YPRqefhoOOggWLky7IhERyaYgIQVvn31g\nzBh47jno1QsWLEi7IhERqaQgIUVhr71g7Fh44QXYf3/49tu0KxIREVCQkCKy554xXuLll6PL4+uv\n065IREQUJKSo7L47PPUUTJ8O++4LX32VdkUiIg2bgoQUnV/9CiZMgNdfh27d4Msv065IRKThUpCQ\norTjjvEkx9tvw957w+efp12RiEjDpCAhRatTJ3jmGXj/fejaFebNS7siEZGGR0FCitr228M//gEf\nfxxPdnz2WdoViYg0LAoSUvS23RaefTZCRJcuMGdO2hWJiDQcChJSErbZBv75zxh4+ZvfwKefpl2R\niEjDoCAhJWPLLSNMfPNNzDnx0UdpVyQiUvoUJKSkbL55hInvv48w8cEHaVckIlLaFCSk5Gy6KUyc\nCO4RJt57L+2KRERKl4KElKSNN46WiSZNYI894J130q5IRKQ0KUhIyWrfPp7mWG21aJmYNSvtikRE\nSo+ChJS0DTaIMLHGGvE0xxtvpF2RiEhpUZCQkrfeejFp1dprR5iYMSPtikRESoeChDQIbdvGdNpt\n28akVa+8knZFIiKlQUFCGox11okwseGGMZ32tGlpVyQiUvwUJKRBWXvtWDV0000jTEyenHZFIiLF\nTUFCGpy11oKnnoqZMLt2hRdfTLsiEZHipSAhDVJZGYwfHwt+desGzz+fdkUiIsVJQUIarDXWgCee\ngI4doXt3eO65tCsSESk+ChLSoLVuDWPHwk47wb77xpwTIiJSewoS0uC1agWPPQa77Qb77QcTJqRd\nkYhI8VCQEAFatoTRo2PCqp49Y/yEiIismIKESEaLFvD3v8Pee0OvXjBmTNoViYgUPgUJkSzNm8PD\nD0cXx8EHw6OPpl2RiEhhSxQkzGywmb1rZgvNbJKZ7VjDsXua2dKcbYmZrZtz3OFmNjNzzelm1iNJ\nbSIrq1kzGDUKDjwQDjssgoWIiFQt7yBhZr2B64CLgY7AdGC8mbWp4TQHtgDaZbb13P2zrGvuCtwH\n/BnoADwKPGJm2+Rbn0hdaNoUKioiSPTuHcFCRER+KkmLxBDgVne/293fAE4EFgD9V3DeXHf/rHLL\nee80YJy7X+/ub7r7MGAKcEqC+kTqRJMm8Ne/Qnl5bPfdl3ZFIiKFJ68gYWZNgc7A05X73N2BCcAu\nNZ0KTDND+q99AAAY6ElEQVSzT8zsyUwLRLZdMtfINn4F1xSpd02awMiRcMwxcPTRcPfdaVckIlJY\nmuR5fBugMTAnZ/8cYMtqzvkUGAS8DDQHBgDPmtlO7l65/mK7aq7ZLs/6ROpc48bwl79Ed8exx8Li\nxdB/Re1vIiINRL5BIm/uPguYlbVrkpltRnSR9FvZ6w8ZMoSysrLl9pWXl1NeXr6ylxb5UaNGcMst\n0UJx/PGwaBEMGpR2VSIiK1ZRUUFFRcVy++bPn19n1883SMwDlgBtc/a3BWbncZ2XgN2yXs9Oes3h\nw4fTqVOnPL61SDKNGsHNN0eYOPHEaJkYPDjtqkREalbVL9dTpkyhc+fOdXL9vMZIuPsiYDLQtXKf\nmVnmdT7rJ3YgujwqvZB9zYxumf0iBcMMbrgBzjgDTjkF/vCHtCsSEUlXkq6N64GRZjaZaFkYArQE\nRgKY2VXA+u7eL/P6dOBdYAbQghgj0YUICpVuIMZNnAGMAcqJQZ0DEtQnUq/M4P/+L8ZMDBkSLRNn\nnZV2VSIi6cg7SLj7qMycEZcS3Q/TgO7uPjdzSDugfdYpzYh5J9YnHhN9Bejq7hOzrvmCmfUBrshs\nbwEHuvvr+d+SSP0zg6uuijBx9tkxZuK889KuSkRk1Us02NLdRwAjqnnvuJzX1wLX1uKaDwOaQ1CK\nhhlcdlmEifPPjzAxbFjaVYmIrFr1/tSGSKkbNiweEb3wwujmuOSSCBkiIg2BgoRIHbjggmiZOOec\naJm48kqFCRFpGBQkROrI0KERJs44I8LEtdcqTIhI6VOQEKlDQ4bEPBOnnRbdHMOHK0yISGlTkBCp\nY6eeGi0TJ50ULRN//GNMZiUiUooUJETqwYknRsvEwIHRMvGnPylMiEhpUpAQqScnnBBhon//CBO3\n3RZPd4iIlBIFCZF6dOyxESb69YtujjvvVJgQkdKiICFSz/r2jTDRt2+0TNx9d7wWESkF+nEmsgoc\neWSEh/LyCBP33hsDMkVEip2Gf4msIocdBg8+CI88Ar17ww8/pF2RiMjKU5AQWYUOOgj+9jcYMyaC\nxfffp12RiMjKUZAQWcUOOCBaJZ58Eg45BL77Lu2KRESSU5AQSUGPHjB6NDzzDBx4ICxcmHZFIiLJ\nKEiIpGSffaKL41//gp49YcGCtCsSEcmfgoRIivbaC8aNg0mTYL/94Jtv0q5IRCQ/evxTJGV77AHj\nx0d3R48eMHYsrL562lWJSLFYtAi++AI+/7z222ef1d33V5AQKQC77RaDL7t3jy6PJ56AsrK0qxKR\nVWnhwtoHgezg8PXXVV+vVSv42c+W3zbaKL4uWBALCtYFBQmRAvGrX8HTT0O3brE9+SSsuWbaVYlI\nPtzjg702ASB3q+4JrjXXXBYE1loL1lkHttzypyEhe1trLWjevPo6p0xRkBApSTvssCxMdO0KTz0V\nPxREJF0zZ8LUqbVrLViy5KfnN2pUdetAhw41B4KyssKfUr/AyxNpeDp1isdC9947BmNOmABt2qRd\nlUjDtHgxXHEFXHZZBIRmzWDttZf/sN9qqxW3Dqy+eoSJUqQgIVKAtt8e/vGPaJWoDBPrrpt2VSIN\ny1tvwdFHw3/+AxdeCGefHeMOzNKurLCUaD4SKX7bbgvPPgtz50KXLjB7dtoViTQM7nDbbdHtMG8e\n/PvfcMkl0Lq1QkRVFCRECtjWW0eY+PJL+M1v4JNP0q5IpLTNmQO9esGgQXDUUTBtWgyEluopSIgU\nuC23hH/+Mx7X2nNP+PDDtCsSKU2PPQbbbQcvvgiPPhqtEq1bp11V4VOQECkCm28eYWLRoggT77+f\ndkUipeObb2DgwGiJ2HlnePXV+G+pHQUJkSKxySYRJiDCxLvvpluPSCl44YUYC3HvvdECMXo0tG2b\ndlXFRUFCpIhsvHGEiaZNI0y8/XbaFYkUp0WLYNgw2H33eLx62jQYMECDKZNQkBApMu3bxwDM1VaL\nMDFrVtoViRSXN9+MaemvvBIuvjhW4N1ii7SrKl4KEiJFaIMNomWirCzCxMyZaVckUvjc4U9/go4d\n40mo55+PVolCnzmy0ClIiBSpdu2iZaJNm3g09LXX0q5IpHDNng0HHAAnnwz9+sV01zvtlHZVpUFB\nQqSIrbtuzIC53noxadX06WlXJFJ4HnkkHuucPBkefzxaJVq1Sruq0qEgIVLk2rSJhb7at4/ptKdO\nTbsikcLw9dfQvz8cfHCMiXj1Vdh//7SrKj0KEiIlYO21I0xstlmEiZdfTrsikXT9+9+xZs2oUXD7\n7fD3v8fy21L3FCRESsRaa8Wy41ttFSuHvvhi2hWJrHo//AAXXAB77BFdftOnw/HH67HO+qQgIVJC\nyspg/PhY8Ktbt/itTKShmDkTdtkFfv97uPTSeLJps83Srqr0KUiIlJg11oAnnoBOnaB7d5g4Me2K\nROqXO9x0U/yb//bbmK3yggv0WOeqoiAhUoJat4YxY2LdgB494Jln0q5IpH588kn8Gz/11OjCmDIF\ndtgh7aoaFgUJkRLVqlWsZrjbbjFS/amn0q5IpG49/HA81jl9OowbF60SLVumXVXDoyAhUsJatoxF\niLp0gZ49o8tDpNh99RUceywcdlhMxvbqq7DvvmlX1XAlChJmNtjM3jWzhWY2ycx2rOV5u5nZIjOb\nkrO/iZkNM7O3M9ecambdk9QmIstr0SIefdtnHzjwQHj00bQrEknuuefisc6//Q1GjoSHHoq5VCQ9\neQcJM+sNXAdcDHQEpgPjzazGv0ozKwPuAiZU8fYVwABgMLA1cCvwdzPbPt/6ROSnmjePH7j77w8H\nHQRdu0ZXh3valYnUzg8/wHnnxdoyG24Y3Rn9+umxzkKQpEViCHCru9/t7m8AJwILgP4rOO8W4F5g\nUhXv9QWucPfx7v6eu98CjAXOTFCfiFShWTN48MHY5s+PFoodd4yAsWRJ2tWJVG/GjBg4fN11sWLn\ns8/CJpukXZVUyitImFlToDPwdOU+d3eilWGXGs47DtgEuKSaQ5oD3+fsWwjsnk99IlKzxo2jX/k/\n/4kWiTXXhMMPh623jtn/vs/9v1AkRUuXwg03QOfO8W/zxRfh3HPj37EUjnxbJNoAjYE5OfvnAO2q\nOsHMtgCuBI5y96XVXHc8cIaZbW6hG3AIsF6e9YlILZjF7JcTJsQP5+22g4ED47e8//u/WKNAJE0f\nfxwDKH/7Wxg0KBbc6tgx7aqkKvX61IaZNSK6My5293cqd1dx6OnAW8AbRMvEjcAdQHXBQ0TqyE47\nxWN0r78ez+Offz5stBFcdBHMnZt2ddIQjRoV4XbGDHjyyWiVWG21tKuS6pjnMdoq07WxADjU3Udn\n7R8JlLn7wTnHlwFfAItZFiAaZf57MbCPuz+bdXwzYG13/9TMrgb2d/ftqqmlEzB5jz32oKysbLn3\nysvLKS8vr/V9icgyH34Iw4fDbbdF0/Lxx8NZZ8HGG6ddmZS6+fPhlFPgnnuiy+2WW+BnP0u7quJX\nUVFBRUXFcvvmz5/PxJj2trO7T6nyxFrKK0gAmNkk4EV3Pz3z2oAPgBvd/dqcY414CiPbYKALcCjw\nnrsvrOJ7NAVeB+5394uqqaMTMHny5Ml06tQpr3sQkRX73//g5pvhxhvhyy+hTx8YOjTW8RCpa//8\nJxxzTPxbu+km6NtXT2TUpylTptC5c2eogyCRpGvjemCAmR1jZlsRT2O0BEYCmNlVZnYXxEBMd389\newM+A75z95mVIcLMdjKzg81sEzP7NTCOaLW49qffXkRWhbXXhmHD4P33Y7T8P/4Rzc29esVaBiJ1\n4fvv4eyzY9K0TTaBV16Bo49WiCgmeQcJdx8FnAVcCkwFfgl0d/fK3tR2QPs8L9sCuByYATwMfAjs\n7u5f5VufiNStVq3g9NPhnXdiAqC334Zdd43n+ceN01wUktyrr8YYnRtugGuugaefVhdaMUo02NLd\nR7j7z919NXffxd1fznrvOHffq4ZzL3H3Tjn7Jrr7L9y9pbuvm7nG7CS1iUj9aNYsJgB67bWYKfO7\n72C//WIk/f33w+LFaVcoxWLpUrj++lhca8mSeBz57LP1WGex0lobIpKXRo1idsxJk6K7o21bKC+H\nLbeMwXHffZd2hVLIPvwQunWDM8+EwYPh5ZdjymspXgoSIpKIWSyYNH58POO/ww7xwfDzn0cz9fz5\naVcohaaiAn75S3jzzZjD5PrrYy0YKW4KEiKy0jp1ggcegDfeiIXBhg2LuSjOOw9mq5Oywfvii3jq\np0+fmGTq1VdjvRcpDQoSIlJnttgCbr0V3nsvZiO86aZooTj5ZPjvf9OuTtLwzDPRCjF2LNx7b7RK\nrLVW2lVJXVKQEJE6t9568PvfwwcfROvEQw9FyOjTJ1ZtlNI3b16Mg+jaNf7uX3kl/v6l9ChIiEi9\nWWutmHL7/fdjYqvnn4cOHWI58+ee06OjpcAd3n03nuQZNgx69oxlvtdZJ1qkrrsuxkNstFHalUp9\naZJ2ASJS+lZbLQZiDhwY6yhcfTXssUfMR3HuuREsGunXmoK3aFGMg5k6FaZNW/b1yy/j/XXXjceB\njz46vu66a4QKKW0KEiKyyjRtCkcdFU3cY8bAVVfFTJnbbgvnnAO9e8cxkr5vv43uiKlTl22vvbZs\nqfnNNouwcNZZ8bVjx+jSkoZHQUJEVjkzOOCA2P71r2ihOPpouPDC+GDq3x9atky7yoZj7tzlWxmm\nToVZs6LbokkT+MUvIigcc0x0TW2/PeSslSgNmIKEiKRq993h8cfjt99rronpuC+9NL6efLJG+Ncl\n93iiJruVYdo0+PjjeL916wgJ3brFAm0dO8I220Dz5qmWLQUu79U/C4VW/xQpTf/9bwzQ+8tfopvj\nxBNhyBBYf/20KysuixbBzJk/Hc9QOVFY27bLuiQ6dIivm22msSoNRV2u/qkgISIFac6cWMzp5ptj\n2u1+/WI9hi22SLuywvPNN8uPZ5g2rerxDNnBQeMZGra6DBLq2hCRgtS2LVx5ZQzCvPVWGD4cbr8d\nDjssnvRoqL8/fPbZ8mMZpk6Ft96KboumTWM8Q4cOMZ6hY8foqlhjjbSrllKmICEiBa2sLPrrTzsN\n7r47Jrrq3Bn22ScCxW9+E4M3S03l/Ay5gyA/+STeb906AkP37hG2NJ5B0qIgISJFoUWLmIeif394\n+OF40mOvveK9Jk1iCeqqtpreq+/38z13yRKYMWNZePjqq7i/du0iNPTrt6x7YtNNNZ5BCoOChIgU\nlSZNYr6JI46IdRzeeSc+gLO3xYt/uq8271X3/uLFMd4gybm1/b6VNt88gsK55y4bBNmuXXp/3iIr\noiAhIkXJLNZxKIVVJN1h6dJl8zaIFBP9kxURSZlZdG2IFCP1sImIiEhiChIiIiKSmIKEiIiIJKYg\nISIiIokpSIiIiEhiChIiIiKSmIKEiIiIJKYgISIiIokpSIiIiEhiChIiIiKSmIKEiIiIJKYgISIi\nIokpSIiIiEhiChIiIiKSmIKEiIiIJKYgISIiIokpSIiIiEhiChIiIiKSmIKEiIiIJKYgISIiIokp\nSIiIiEhiChIiIiKSmIJEgaioqEi7hDql+ylcpXQvoPspZKV0L1B691NXEgUJMxtsZu+a2UIzm2Rm\nO9byvN3MbJGZTanivTPN7E0zW2BmH5jZ9WbWPEl9xajU/oHqfgpXKd0L6H4KWSndC5Te/dSVvIOE\nmfUGrgMuBjoC04HxZtZmBeeVAXcBE6p47xjgcmAYsBXQHzgCuCLf+kRERGTVSdIiMQS41d3vdvc3\ngBOBBcSHf01uAe4FJlXx3o7Av9z9AXf/wN0nAPcDOyWoT0RERFaRvIKEmTUFOgNPV+5zdydaGXap\n4bzjgE2AS6o55AmgY2UXiZltCuwHjMmnPhEREVm1muR5fBugMTAnZ/8cYMuqTjCzLYArgd3dfamZ\n/eQYdx9jZhcDz5uZZ77HLe5+TQ21tACYOXNmnrdQmObPn8+UKT8ZOlK0dD+Fq5TuBXQ/hayU7gVK\n636yPjtbrPTF3L3WG7AesBTYOWf/NcALVRzfCHgJGJi173fAlJzjDga+BI4DfgEcCLwPXFhDLX0A\n16ZNmzZt2rQl3vrkkwOq2izzoVwrma6NBcCh7j46a/9IoMzdD845vgz4AlgMVDZFNMr892JgH3d/\n1swmARPdfWjWuUcRYzFaV1PL2kB34D3gu1rfhIiIiLQAfg6Md/f/rcyF8uracPdFZjYZ6AqMBrDo\nq+gK3FjFKV8B2+bsGwx0AQ4lQgBEuFicc9zSyut7FWknc+P35VO/iIiI/Oj5urhIvmMkAK4HRmYC\nxUvEUxwtgZEAZnYVsL6798sEgNezTzazz4Dv3D17cMMjwFAzmw68CGwBXAqMripEiIiISGHIO0i4\n+6jMnBGXAm2BaUB3d5+bOaQd0D7Py16d+XopsCEwl2jxuDDf+kRERGTVyWuMhIiIiEg2rbUhIiIi\niSlIiIiISGJFFyTM7NdmNtrMPjazpWbWK+2akjKz88zsJTP7yszmmNnfzez/pV1XUmZ2oplNN7P5\nme15M9s37brqgpmdm/n3dn3atSRhZhdn6s/eXl/xmYXJzNY3s7+a2bzMQn/TzaxT2nUlkVkAMffv\nZqmZ/THt2pIws8ZmdlXmvhaY2dtmVrTj3cystZn9wczey9zPv8xsh7Trqo3afF6a2aVm9knm3p4y\ns83z/T5FFySAVsQAz5OJyTSK2a+BPwI7A3sDTYEnzWy1VKtK7kPgHKATMZX6M8BoM9sm1apWUmbq\n9oHEAnXF7DVigHS7zLZ7uuUkY2ZrAv8GvifmktkaOJOYs6YY7cCyv5N2QDfiZ9uoNItaCRcAxwMn\nEYswDiWeyjsl1aqS+wsxxcFRxHQGTwETzGy9VKuqnRo/L83sHOAU4ufbTsC3xCKczfL5JkU92NLM\nlgIHZU+OVcwyT8N8Buzh7v9Ku566YGb/A85y9zvTriUJM2sNTCZ+KF4ETHX3M9KtKn+ZKegPdPei\n/K09m5ldDezi7numXUt9MLM/APu5e1G2TprZY8Bsdx+Qte8hYIG7H5NeZfkzsxbA10BPd38ia//L\nwFh3H5ZacXmq6vPSzD4BrnX34ZnXaxBLXvRz91oH2WJskShlaxKp8fO0C1lZZtbIzI4EmgPPpV3P\nSrgZeMzdn0m7kDqwRaaJ8x0zu8fM8n1Mu1D0BF42s1GZLsEpZnZC2kXVhczswUcRvwUXq3FA18w6\nS5jZ9sBuwNhUq0qmCbH20/c5+xdSpC16lcxsE6IFLHsRzq+IuZyqXYSzKkkmpJJ6kJkh9A/EcurF\n3He9LfACMf3qAuAId3873aqSyQShDkTTc7GbBBwLvEmsmfM7YKKZbevu36ZYVxKbEi1E1wFXEE2y\nN5rZ9+7+11QrW3kHA2XAXWkXkpS7j8iE1DfNbDHxC+sF7n5/yqXlzd2/MbMXgIvM7A3it/U+xAft\nW6kWt/LaEb+4VrUIZ7t8LqQgUThGANsQyb2YvQFsT/wwPAy438z2dPep6ZaVHzPbkAh2e7v7orTr\nWVnuPj7r5Wtm9hKxMN4RQLF1OzUCXnL3izKvp2cC7IlAsQeJ/sA4d5+ddiFJmdlpQD+gNzGzcQfg\nBjP7pEiDXl/gDuBjYimHKcTyDJ3TLKqQKEgUADO7CdgP+LW7f5p2PSvD3RcD/828nGpmOxG/PQ5M\nr6pEOgPrAFMyrUUQTZx7ZAaNNS/m6dvdfb6ZzQLyHqFdAD4FZubsmwkckkItdcbMNiIGXR+Udi0r\n6XzgEnd/MPN6hpn9HDiPIgx67v4u0CUzCH4Nd59jZvez7OdcsZpNLKDZluVbJdoCef3ipzESKcuE\niAOBLu7+Qdr11INGxAdwsZkAbEf8NrV9ZnsZuAfYvphDBPw4iHRz4kO52Pwb2DJn35ZEC0sx60/8\nQC/GsQTZGgFLcvYtpcg/b9x9YSZErEU8LfRI2jWtjExAmk08kQL8ONhyZ/JczKvoWiTMrBXxA7Dy\nt8RNM4N5Pnf3D9OrLH9mNgIoB3oB35pZ28xb89296JZGN7MriYFWHwCrE4PG9gAuT7OuJDLjBnIX\nnPsW+F/OgnNFwcyuBR4jPmw3AC4BFgEVadaV0HDg32Z2HvGI5M7ACcCAGs8qYJlWr2OBke6+NOVy\nVtYjwIVm9hEwg3gcfAhwe6pVJWRm+xCfN28SC0r+nvjZMDLFsmqlFp+XfyD+rt4mVuO+DPgIeDSv\nb+TuRbUBexLpdknOdkfatSW4l6ruYwlwTNq1Jbyf24nmvoVE0n0S2Cvtuurw/p4Brk+7joS1V2R+\nQCwkgt59wCZp17US97Mf8AoxoHcG0D/tmlbyfrpl/t/fPO1a6uBeWgLXZn4WfEsMSrwEaJJ2bQnv\n53Dg7cz/Ox8DNwCrp11XLWtf4eclMfD6k8z/S+OT/Bss6nkkREREJF1F3WclIiIi6VKQEBERkcQU\nJERERCQxBQkRERFJTEFCREREElOQEBERkcQUJERERCQxBQkRERFJTEFCREREElOQEBERkcQUJERE\nRCSx/w+D+SVDtM7k+gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10e003860>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.plot(params, test_scores)\n",
    "plt.title(\"Param vs CV Error\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "大概6~7的时候达到了最优解"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: 上传结果\n",
    "\n",
    "用我们测试出的最优解建立模型，并跑跑测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rf = RandomForestRegressor(n_estimators=30, max_depth=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=6,\n",
       "           max_features='auto', max_leaf_nodes=None,\n",
       "           min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "           min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "           n_estimators=30, n_jobs=1, oob_score=False, random_state=None,\n",
       "           verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_pred = rf.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "把拿到的结果，放进PD，做成CSV上传："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.DataFrame({\"id\": test_ids, \"relevance\": y_pred}).to_csv('submission.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结：\n",
    "\n",
    "这一篇教程中，虽然都是用的最简单的方法，但是基本框架是很完整的。\n",
    "\n",
    "同学们可以尝试修改/调试/升级的部分是：\n",
    "\n",
    "1. **文本预处理步骤**: 你可以使用很多不同的方法来使得文本数据变得更加清洁\n",
    "\n",
    "2. **自制的特征**: 相处更多的特征值表达方法（关键词全段重合数量，重合比率，等等）\n",
    "\n",
    "3. **更好的回归模型**: 根据之前的课讲的Ensemble方法，把分类器提升到极致"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
